Testing Task Force Quarterly Update

This article originally appeared in The Bar Examiner print edition, Fall 2019 (Vol. 88, No. 3), pp 45–46.

For the latest Testing Task Force news—and to sign up to receive updates with the latest Task Force news, research, and blog posts—visit www.testingtaskforce.org.Testing Task Force Nationwide 2019 Practice Analysis Concludes

The Testing Task Force started this fall season with a big win: it successfully concluded its nationwide practice analysis—the core work in Phase 2 of its three-year study—in late September, with over 30,000 people accessing the survey. The practice analysis asked newly licensed lawyers and attorneys who have, or have had, direct experience working with newly licensed lawyers to rate lists of job tasks, KSAOs (knowledge, skills, abilities, and other characteristics), and/or technologies. The job tasks were grouped into the four broad categories of General, Litigation/Dispute Resolution, Transactional, and Regulatory/Compliance but were otherwise practice-area neutral so that the data will highlight general work activities that all newly licensed lawyers perform regardless of their practice specialties.

The results from the thousands of ratings will help to identify core job tasks, KSAOs, and technologies that are important for all newly licensed lawyers. The responses to the demographic questions included at the end of the survey will also allow the Task Force to analyze the ratings of tasks, KSAOs, and technologies by groups of respondents and to highlight trends based on such demographics as region, practice setting, and years of experience.

The Task Force received assistance from bar associations and state Supreme Courts across 54 jurisdictions to let attorneys around the country know about the opportunity to participate in the practice analysis survey. We extend a huge thank-you to the bar associations and Courts for their assistance. The practice analysis survey could not have been a success without them.

ACS Ventures LLC, the independent psychometric consulting firm that conducted the practice analysis survey, is currently analyzing the data, and the Task Force is eagerly waiting to learn what the results will reveal. A comprehensive report on the practice analysis will be published at the beginning of 2020. The Phase 2 results will be informative regarding the content that should be considered for inclusion on the next generation of the bar exam and will provide empirical data on the competencies newly licensed lawyers need and the work they perform, which should be of interest to all stakeholders.

Phase 1 Report Available Online

Besides working to launch the practice analysis, the Task Force was also busy over the summer writing its Phase 1 report, Your Voice: Stakeholder Thoughts About the Bar Exam, which was published in August. The report summarizes the comments made at the 30 stakeholder listening sessions that were held from November 2018 through June 2019.

The purpose of the sessions was to gather information from various stakeholder groups about their views on the current bar exam to inform exam redesign options, which the Task Force will consider in Phase 3 of its study. Over 400 people participated in the sessions, which were mainly conducted in person in conjunction with scheduled stakeholder events and complemented by a few virtual sessions:

• Council of Bar Admission Administrators Fall Meeting, Denver, CO, November 2018
• Association of American Law Schools Annual Meeting, New Orleans, LA, January 2019
• UBE Jurisdictions Forum, San Antonio, TX, January 2019
• ABA Midyear Meeting, Las Vegas, NV, January 2019
• LSAC/NCBE Legal Educators Conference, Albuquerque, NM, February 2019
• NCBE Annual Bar Admissions Conference, San Francisco, CA, May 2019
• ABA Diversity and Inclusion Center and Pipeline Council, Virtual, May 2019
• Association of Academic Support Educators Conference, Seattle, WA, May 2019
• ABA Deans Workshop, Denver, CO, June 2019
• ABA Young Lawyers Division, Virtual, June 2019

Phase 1 Report Takeaways

The main takeaways from the stakeholder listening sessions can be summarized as follows:

• People favor performance tests (because they are viewed as being representative of what newly licensed lawyers do) and tend to prefer constructed-response items over multiple-choice items.
• They generally consider the Multistate Bar Examination (MBE) to be testing core legal subjects that newly licensed lawyers need to know, but they believe that the MBE tests those subjects too deeply and that, overall, too many topics are tested on the exam.
• Emphasizing lawyering skills over subject-matter knowledge on the next generation of the exam was repeatedly mentioned.
• There were also suggestions to add a component to test legal research skills by using technology to provide examinees with a closed universe of legal resources.
• Regarding timing of the exam, the notion of step testing came up frequently as offering advantages, such as testing MBE subjects closer in time to when law students complete those courses in law school, but the downsides of step testing were also mentioned, such as taking time away from summer employment opportunities.

Besides those themes, a lot of additional ideas were mentioned during the sessions, and we thank all the participants who took the time to provide their input.

The full report is available at www.TestingTaskForce.org.

The Last Phase of the Testing Task Force’s Study

The Testing Task Force has the qualitative data from Phase 1 and the quantitative data from Phase 2 that it needs to start the final and most challenging part of the study. During Phase 3, recommendations will be developed about what content should be tested and how it should be tested. Be assured that the Task Force will be involving stakeholders in Phase 3. It plans to complete this phase by fall 2020 and will then prepare a comprehensive report and set of recommendations to be submitted to NCBE’s Board of Trustees in early 2021.

Contact us to request a pdf file of the original article as it appeared in the print edition.